09/720572 



2 414Rac'dPCT/PTO 2 0 DEC 2000: 

S Attorney Docket No. SWIN 2166 

c Express Mail 

m Label No: ET215672f68 

^ Date of Deposit: .« 

o 

CHAPTER II 

TRANSMITTAL LETTER TO THE UNITED STATES ELECTED 

OFFICE (EO/US) 

(ENTRY INTO U.S. NATIONAL PHASE UNDER CHAPTER II) 
P(rr/^RQQ/niQQ7 lung 1QQ Q OR qq\ ?7 . lime 1PPfl (?7 06 98) 

INTERNATIONAL APPLICATION NO. INTERNATIONAL FILING DATE PRIORITY DATE CLAIMED 

mi r>i ir iM Af^F Pink-i ip apparati IS ■ 

TITLE OF INVENTION 

^rohom FINN AYSON 

APPLICANT(S) 

Assistant Commissioner for Patents 
Box PCT 

Washington, D.C. 20231 
Attention: EO/US 

1 . Applicant herewith submits to the United States Elected Office (EO/US) the 
following items under 35 USC 371 : 

A. [x] This express request to immediately begin national 

examination procedures (35 USC 371(f)). 

B. [x] The U.S. National Fee (35 USC 371(c)(1) and other fees (37 

CFR 1 .492) indicated in the attached fee calculation sheet. 
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[x] A copy of the International application as filed [35 USC 371(c)(2)]: 

a. [ ] is transmitted herewith. 

b. [ ] is not required as the application was filed with the United States 
Receiving Office. 

c. [x] has been transmitted 

i. [x] by the International Bureau. Date of mailing of the 
application (from form PCT/IB/308): 06 January 2000 
(06.01.00) . 

ii. [ ] by applicant on (date) . 



[x] A translation of the International application into the English language [35 
USC 371(c)(2)]: 

a. [ ] is transmitted herewith. 

b. [x] is not required as the application was filed in English. 

c. [ ] was previously transmitted by applicant on (date) 



d. [ ] will follow (within 32 months of earliest priority date). 

[x] Amendments to the claims of the International application under PCT Article 
19 [35 USC 371(c)(3)]: 

a. [ ] are transmitted herewith. 

b. [ ] have been transmitted 

i. [ ] by the International Bureau. Date of mailing of the 

amendment (from form PCT/IB/308): . 



ii. [ ] by applicant on (date) 



[x] have not been transmitted as 

i. [x] applicant chose not to make amendments under PCT 
Article 19. Date of mailing of Search Report (from form 
PCT/ISA/210): 31 January 2000 (21.01.00) . 

ii. [ ] the time limit for the submission of amendments has not yet 
expired. The amendments or a statement that amendments 
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have not been made will be transmitted 
before the expiration of the time limit under 
PCT Rule 46.1. 

5. [x] A translation of the amendments to the claims under PCT Article 19 [35 

USC 371(c)(3)]: 

a. [ ] is transmitted herewith. 

b. [ ] is not required as the amendments were made in the English 
language. 

c. [x] has not been transmitted for reasons indicated at point 4c above. 

6. [x] A copy of the International Preliminary Examination Report (PCT/I PEA/409) 

a. [x] is transmitted herewith. 

b. [ ] is not required as the application was filed with the United States 
Receiving Office. 

7. [x] Annex(es) to the International Preliminary Examination Report 

a. [x] is/are transmitted herewith. 

b. [ ] is/are not required as the application was filed with the United 
States Receiving Office. 

c. [ ] is/are not being transmitted as there is/are no Annex(es). 

8. [x] A translation of the annexes to the International Preliminary Examination 

Report 

a. [ ] is transmitted herewith. 

b. [x] is not required as the annexes are in the English language. 

c. [ ] is not being transmitted for the reason indicated at point 7c above. 

9. [x] An oath or declaration of the inventor [35 USC 371(c)(4)] complying with 35 

USC 115 

a. [ ] was previously submitted by applicant on (date) 

b. [ ] is submitted herewith and such oath or declaration 
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i. [ ] is attached to the application 

ii. [ ] identifies the application and any amendments under PCT 
Article 19 which were transmitted as stated in points 4a or b; 
and states that they were reviewed by the inventor as required 
by 37 CFR 1,70. 

c. [x] will be provided in response to a Notice to File Missing 
Requirements. 

10. [x] An International Search Report (PCT/ISA/210) or Declaration under 

PCT Article 17(2)(a): 

a. [ ] is transmitted herewith. 

b. [x] has been transmitted by the International Bureau. Date of mailing 
(from form PCT/IB/308): 06 January 2000 (06.01.00) . 

c. [ ] is not required as the application was searched by the United 
States International Searching Authority. 

d. [ ] will be transmitted promptly upon request. 

e. [ ] has been submitted by applicant on (date) . 

11. [ ] An Information Disclosure Statement under 37 CFR 1.97 and 1.98: 

a. [ ] is transmitted herewith. 
Also transmitted herewith is 
[] Form PTO-1449 

[ ] Copies of citations listed 

b. [ ] will be transmitted within THREE MONTHS of the date of 
submission of requirements under 35 USC 371(c). 

c. [ ] was previously submitted by applicant on (date) 



12. [ ] An assignment document is transmitted herewith for recording. A separate 
[ ] "RECORDATION COVER SHEET" is also attached. 
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13. [x] Additional documents 

a. [ ] Copy of request (PCT/RO/1 01 ) 

b. [x] International Publication No. WO 00/01164 

i. [ ] Specification, claims and drawing 

ii. [x] Front page only 

c. [x] Preliminary amendment 

d. [ ] Verified Statement Claiming Small Entity Status 

e. [x] Abstract 

f. [ ] Other 



14. [x] The above checked items are being transmitted 

a. [x] before 30 months from any claimed priority date. 

b. [ ] after 30 months but before 32 months (surcharge and/or 
processing fee included) from any claimed priority date. 

15. [ ] Certain requirements under 35 USC 371 were previously submitted by the 

applicant on , namely: 



Johrp Smith-Hill 
Reg. No. 27,730 



SMITH-HILL & BEDELL, P.C. 
12670 N.W. Barnes Road, Suite 104 
Portland, Oregon 97229 

(503) 574-3100 
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PATENT 

IN THE UNITED STATES PATENT AND TRADEMARK OFFICE 
In re Application of 

Graham F INLAY SON Art Unit: 

Application No: Examiner: 
Filed: 

For: COLOUR IMAGE PICKUP APPARATUS 

PRELIMINARY AMENDMENT 

Commissioner for Patents 
O Washington, D.C. 2 0231 

Cj Sir: 

ji: Please make the following amendments to this application prior 

|Je to examination thereof. 

■* ? AMENDMENTS 
O In the Claims : 

W Claims 3-30, 33-44 and 48-56, cancel, 

lt\ Add new claims as follows: 

pj 57. Image recording apparatus (10) according to claim 1 

N 5 wherein first and second optical sensor means (12) are provided for 
recording the first and second images respectively. 

58. Image recording apparatus (10) according to claim 57 
wherein at least one of the first and second optical sensor means 

(12) is relatively broadvand optical sensor means, being responsive 
to at least two distinct wavelengths of light within a broad 
spectrum of wavelengths . 

59. Image recording apparatus (10) according to claim 58, 
wherein the wavelengths are at least 100 nm apart. 

60. Image recording apparatus (10) according to claim 1 
wherein the optical sensor means (12) includes at least two types of 
optically sensitive elements responsive to respectively different 
wavelengths of light. 
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61. Image recording apparatus according to claim 1 wherein 
the optical sensor means (12) comprises a charge coupled device 
(CCD) chip, the chip comprising an array of photoelectric detector 
pixels . 

62. Image recording apparatus according to claim 61 wherein 
the pixels have a broad response centering on a particular 
wavelength of light. 

63. Image recording apparatus according to claim 61 wherein 
the CCD chip is coated with a filter (14) . 

64. Image recording apparatus according to claim 1 wherein 
the optical processing means comprises an optical filter (14) . 

65. Image recording apparatus according to claim 64 wherein 
the filter (14) has characteristics such that its output is linearly 
related to its input. 

66. Image recording apparatus according to claim 64 wherein 
the response of the filter (14) is a smooth function with respect to 
wavelength and the filter (14) has an average transmi ttance of more 
than 30%. 

67 . Image recording apparatus according to claim 64 wherein 
the filter (14) produces an output which includes relatively more 
light of one wavelength than of another wavelength as compared with 
the input . 

68. Image recording apparatus according to claim 64 wherein 
the filter (14) is located in the image light path before the 
optical sensor means. 

69. Image recording apparatus according to claim 1 wherein 
first and second optical sensor means are provided by a single CCD 
chip (12) which records the first and second digital optical images. 

70. Image recording apparatus according to claim 69 wherein 
the first and second sensor means may compise respectively different 
parts of the chip (12) . 
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71. Image recording apparatus according to claim 69 wherein 
the first and second images comprise different parts of the image 
recorded by the CCD chip (12) in spatial terms or in terms of the 
frequencies of light recorded. 

72. Image recording apparatus according to claim 69 wherein a 
filter (14) is provided in front of or on a part of the CCD chip 
(12) such that the first or second digital optical image is recorded 
by that part of the chip, and the other of the digital optical 
images is recorded by the remainder of the chip. 

73 . Image recording apparatus according to claim 1 wherein 
the optical sensor means comprises a first CCD chip (12a) for 
recording the first digital optical image and a second CCD chip 
(12b) for recording the second digital optical image. 

74 . Image recording apparatus according to claim 73 wherein 
the chips (12a, 12b) are located in close proximity to one another, 
in the same geometric plane. 

75. Image recording apparatus according to claim 73 wherein 
the two CCD chips (12a, 12B) are responsive to respectively 
substantially the same frequencies of light, the optical processing 
means comprising an optical beamsplitter for splitting the image 
light into two parts and for directing each part of the light 
towards a respective one of the CCD chips, and an optical filter 
(14) being located in the path of one part of the image light, 
before one CCD chip . 

76. Image recording apparatus according to claim 73 wherein 
the optical sensor means and the optical processing means are 
located within a housing, such as a camera body. 

77. Image recording apparatus according to claim 73 wherein 
each CCD chip (12a, 12b) is provided in a separate housing, a first 
housing having a CCD chip (12b) provided therein and a second 
housing having a CCD chip (12a) and an optical filter (14) provided 
therein . 

78. Image recording apparatus according to claim 73 wherein a 
first CCD chip is provided within a first digital camera and a 



second CCD chip is provided within a second digital camera, such 
that the different optical processing of the two images results from 
the different camera characteristics. 

79. Image recording apparatus according to calim 73 wherein 
the two chips are responsive to respectively different frequencies 
of light. 

80. Image recording apparatus according to claim 1 wherein 
the processing means is microprocessor based, having electrical 
memory means . 

81. Image recording apparatus according to claim 1 wherein 
the processing means includes means for providing information 
relating to the spectral characteristics of the illuminant light. 

82. Image recording apparatus according to claim 81 wherein 
information relating to the spectral characteristics of the 
illuminant light is used to facilitate removal of at least some of 
any illuminant colour bias present in the recorded image. 

83. Image recording apparatus according to claim 81 wherein 
the processing means includes means for facilitating the removal of 
at least some of any demosaicing errors and/or interref lection 
errors and/ or shadows present in the recorded image. 

84. Image recording apparatus according to claim 81 wherin 
the processing means includes means for providing information 
relating to the physics of the scene, such as the physical 
characteristics of the scene . 

85. A method for recording an image according to claim 31 
wherein different optical processing results at least partly from 
the filtering of light producing the first or second image. 

86. A method according to claim 31 wherein the different 
optical processing is provided by the use of sensors responsive to 
respectively different frequencies of light in recording the fist 
and second images . 



87. A method for recording an image according to claim 31 
wherein the first and second images comprise respectively different 
parts of a global image of a scene. 

88. A method for recording an image according to claim 31 
wherein the processing of the information relating to the first and 
second images provides an estimate of the spectral characteristics 
of the illuminant light. 

89. A method for calibrating image recording apparatus, the 
method being according to claim 31. 

90. A method according to claim 89 wherein the method 
includes the carrying out of steps (a) and (b) for each of a 
plurality of different known illuminant lights and wherein step (b) 
includes the step of processing the information relating to the 
first adn second images to provide an indication of the relationship 
therebetween . 

91. A method accoridng to claim 90 wherein the indication of 
the relationship is a transform function, which may be a transform 
matrix, and the method provides a set of reference transform 
functions, each transform function relating to a different known 

i 1 luminant 1 ight . 

92. A method according to claim 31 for processing an image 
recorded using image recording apparatus wherein the first and 
second images relate to a scene illuminated by an unknown 
illuminant . 

93 . A method according to claim 92 wherein the method 
includes the step of applying one or more of the reference transform 
functions to the first or second image and determining the reference 
transform function which best relates the two images. 

94 . A method according to claim 93 wherein each reference 
transform function is applied to the first image to produce a 
transformed first image, which is subsequently compared to the 
second image and the reference transform function which produces a 
transformed first image most closely resembling the second image is 
selected as the best reference transform function. 



95. A method according to claim 94 wherein the known 
illuminant light to which the best reference transform function 
relates is determined, to provide information relating to the 
spectral characteristics of the light illuminating the scene to be 
recorded . 

96. A method according to claim 95 hwerein at least some of 
the colour bias due to the illuminating light is removed from the 
image of the scene to be recorded and/or at least some demosaicing 
errors and/or interref lection errors and/or shadows present in the 
recorded image are removed . 

97. A method according to claim 45, the method further 
including the following image recording steps: 

recording the responses of the image recording apparatus to 
image light (P t ) from a scene to be recorded and to optically 
filtered image light (P 2 ) from the scene; 

determining which colour of illuminant E(X) is closest to the 
colour of the illuminating light, thereby estimating the colour of 
the illujminating light; and 

removing at least some colour bias due to the illuminating 
light from the recorded image and/or at least some of any 
demosaicing errors and/or interref lection errors. 

98. A method according to claim 97 wherein the optically 
filtered image light is filtered using a filter which produces an 
output which includes relatively more light of one wavelength than 
the input . 

99. A method according to claim 97 wherein the colour of the 
illuminating light is determined by applying each transform function 
T a - b to the recorded response {P ± ) of the apparatus to the image 
light and comparing the transformed response (Pi) to the response 
(P 2 ) of the apparatus to the filtered iamge light, the transform 
function which best relates the two responses being the function 
which identifies the colour of the image light and the filtered 
image light. 

100. A method according to claim 99 wherein the best transform 
function is defined as the function which minimizes the error of the 
operation (T a,b P L - P 2 ) . 



REMARKS 

The above amendments are presented in order to place this 
application in better condition for examination. 

Respectfully submitted, 



SMITH-HILL & BEDELL, P.C. 

12670 N.W. Barnes Road, Suite 104 

Portland, Oregon 97229 

Tel. (503) 574-3100 
Fax (503) 574-3197 

Docket: SWIN 2166 
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The present invention relates to image recording apparatus for 
processing an image. The invention also relates to a method for recording an 
image with image recording apparatus and processing the recorded image. 

An image here refers to a record of a part or the whole of a scene to be 
recorded; the image may comprise a part or the whole of the record of the 
scene. An image may comprise a part of a scene in spatial terms. For example, 
an image may comprise only a central area of a recorded scene. Further, an 
image may comprise part of a scene in terms of the frequencies of light 
recorded. For example a first image of a scene might record mainly red light 
while a second image of the same scene records mainly green light. 

An image recorded by a camera, or by computer vision, remote sensing 
or image processing systems, and the like is generally dependent upon the 
colour of the light illuminating the scene to be recorded. An image may also be 
affected by the shape of an object in the scene, by shadows and reflections in 
the scene, or by the camera or system itself. It is desirable that any effect that 
these parameters have on an image be known. Knowledge of these effects can 
assist in the removal of colour bias from an image and can provide important 
information concerning the physical characteristics of the scene being viewed. 

According to the invention there is provided image recording apparatus 
for processing an image, the apparatus including: 

optical sensor means for recording a first digital optical image of at least 
a part of a scene illuminated by an illuminant light and for recording a second 
digital optical image of at least a part of substantially the same scene under 
substantially the same illuminant light; 

the light producing the first and second images undergoing different 
optical processing; and 

means in communication with the optical sensor means for processing 
information relating to the first and second images. 
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Preferably the processing means relates one of the first and second 
images to the other of the first and second images. 

Preferably the processing means correlates the first and second images. 

First and second optical sensor means may be provided for recording the 
first and second images respectively. 

Preferably at least one of the first and second optical sensor means is 
relatively broadband optical sensor means. Preferably the broadband optical 
sensor means are responsive to at least two distinct wavelengths of light within 
a broad spectrum of wavelengths. Preferably the wavelengths are at least 50 
nm apart and preferably at least 100 nm apart. They may be as much as 150 
nm apart. Preferably the broad spectrum is at least 100 nm wide and preferably 
at least 200 nm wide. It may be as much as 300 nm wide. The optical sensor 
means may include at least two types of optically sensitive elements, responsive 
to respectively different wavelengths of light. 

The other of the first and second optical sensor means may be relatively 
narrow band, being particularly responsive to wavelengths of light within a 
spectrum less than 100 nm wide. 

However, it is preferred that both of the first and second optical sensor 
means are relatively broadband optical sensor means. 

The optical sensor means preferably comprises a charge coupled device 
(CCD) chip, the chip comprising an array of photoelectric detector pixels. The 
pixels preferably have a broad response centering on a particular wavelength of 
light. The response may be to red, green or blue light. The particular response 
of the CCD chip may be achieved by it being coated with a filter. Alternatively, 
the optical sensor means may comprise a CMOS device. Where a CCD chip is 
referred to throughout the specification, it should be understood that a CMOS 
device could alternatively be used. 
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The optical processing means preferably comprises an optical filter. The 
filter preferably has characteristics such that its output is linearly related to its 
input. Preferably the response of the filter is a smooth function with respect to 
wavelength. Desirably the filter is highly transmitting and may have an average 
transmittance of more than 10%, preferably more than 20% and most preferably 
more than 30%. The filter preferably produces an output which includes 
relatively more light of one wavelength than of another wavelength as 
compared with the input. The output may contain relatively more green light 
or more purple light than the input. The filter is preferably located in the 
image light path before the optical sensor means. The filter may be provided 
on or in front of the optical sensor means. The filter may be provided as a 
coating on the optical sensor means. 

The first and second optical sensor means may be provided by a single 
CCD chip which records the first and second digital optical images. The first 
and second sensor means may comprise respectively different parts of the chip. 
The first and second images may comprise different parts of the image 
recorded by the CCD chip, in spatial terms or in terms of the frequencies of 
light recorded. The parts of the image may be discrete or may overlap. 

The filter may be provided in front of or on a part of the CCD chip such 
that the first or second digital optical image is recorded by that part of the chip, 
and the other of the digital optical images is recorded by the remainder of the 
chip. Preferably the filter is provided in front of or on a central part of the CCD 
chip. The part of the CCD chip preferably represents between l°-6° of its total 
visual field. The optical sensor means and the optical processing means are 
preferably located within a housing, such as a camera body. 

Alternatively, the first image may comprise certain frequencies of light 
recorded by the CCD chip and the second image may comprise different 
frequencies of light recorded by the CCD chip. The frequencies may overlap. 

In another embodiment of the invention, the optical sensor means 
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advantageously comprises a first CCD chip for recording the first digital optical 
image and a second CCD chip for recording the second digital optical image. 
Preferably the chips are located in close proximity to one another. The chips 
are desirably located in the same geometric plane. 

The two CCD chips may be responsive to respectively substantially the 
same frequencies of light. The optical processing means preferably comprises 
an optical beamsplitter for splitting the image light into two parts and for 
directing each part of the light towards a respective one of the CCD chips, and 
an optical filter, the filter being located in the path of one part of the image 
light, before one CCD chip. The filter may be provided on the one CCD chip. 
The optical sensor means and the optical processing means are preferably 
located within a housing, such as a camera body. 

In another embodiment of the invention, each CCD chip is 
advantageously provided in a separate housing, a first housing having a CCD 
chip provided therein and a second housing having a CCD chip and an optical 
filter provided therein. 

In a further embodiment of the invention, a first CCD chip is provided 
within a first digital camera and a second CCD chip is provided within a second 
digital camera, such that the different optical processing of the two images 
results from the different camera characteristics. In this embodiment, a filter is 
not necessarily required. 

The two chips may be responsive to respectively different frequencies of 
light. In this case, the different optical processing results from the different 
responses of the chips and a filter is not necessarily required. 

The processing means is preferably microprocessor based, having 
electrical memory means. The processing means may be a computer. 



The processing means may include means for providing information 
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relating to the spectral characteristics of the illuminant light. 

The information relating to the spectral characteristics of the illuminant 
light may be used to facilitate removal of at least some of any illuminant colour 
bias present in the recorded image. The processing means may alternatively or 
additionally include means for facilitating the removal of at least some of any 
demosaicing errors and/or interreflection errors and/or shadows present in the 
recorded image- The processing means may further additionally or 
alternatively include means for providing information relating to the physics of 
the scene, such as the physical characteristics of the scene. 

According to the invention, there is also provided a method for recording 
an image, the method including the steps of: 

(a) recording a first digital optical image of at least a part of a scene 
illuminated by an illuminant light and recording a second digital optical image 
of at least a part of substantially the same scene illuminated by substantially 
the same illuminant light; 

the light producing the first and second images undergoing different 
optical processing; and 

(b) processing information relating to the first and second images. 

Preferably the processing step includes relating one of the first and 
second images to the other of the first and second images. Preferably the first 
and second images are correlated. 

The different optical processing may result at least partly from the 
filtering of light producing the first or second image. 

Alternatively the different optical processing may be provided by the use 
of sensors responsive to respectively different frequencies of light in recording 
the first and second images. 

The first and second images may comprise respectively different parts of 
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a global image of a scene. The parts may differ in spacial terms or in terms of 
the frequencies of light recorded. The parts of the image may be discrete or 
may overlap. 

Where the first and second images comprise respectively different parts 
of an image in spatial terms, the light producing one of the first and second 
images is preferably filtered. 

The first image may comprise a record of certain frequencies of light 
while the second image comprises a record of respectively different frequencies. 
The frequencies may overlap. The light for both images may originate from the 
same scene. In this case a filter is not necessarily used. 

Alternatively the first and second images may comprise a record of 
respectively substantially the same scene or part of a scene. The first and 
second images may comprise records of respectively substantially the same 
frequencies. In this case, preferably the light producing the first image or the 
second image is optically filtered. 

Preferably the processing of the information relating to the first and 
second images provides an estimate of the spectral characteristics of the 
illuminant light. 

The method may be used for calibrating image recording apparatus 
wherein steps (a) and (b) are carried out for each of a plurality of different 
known illuminant lights. Preferably step (b) includes the step of processing the 
information relating to the first and second images to provide an indication of 
the relationship therebetween. The indication of the relationship is preferably a 
transform function, which may be a transform matrix. Preferably the 
calibration provides a set of reference transform functions, each transform 
function relating to a different known illuminant light. The set may consist of 
at least 10 transform functions, and preferably at least 20 transform functions. 
Alternatively, the calibration may provide a parameterized continuous set of 
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transform functions. 

Preferably the reference transform functions are recorded in an electrical 
memory means. 

The method may also be used for processing an image recorded using 
image recording apparatus wherein the first and second images relate to a scene 
illuminated by an unknown illuminant. 

The processing of the recorded image may include the step of applying 
one or more of the reference transform functions to the first or second image. 
Preferably the reference transform function which best relates the two images is 
determined. Each reference transform function may be applied to the first 
image to produce a transformed first image, which is subsequently compared to 
the second image. Preferably the reference transform function which produces 
a transformed first image most closely resembling the second image is selected 
as the best reference transform function. Preferably the known illuminant light 
to which the best reference transform function relates is determined, to provide 
information relating to the spectral characteristics of the light illuminating the 
scene to be recorded. At least some of the colour bias due to the illuminating 
light may be removed from the image of the scene to be recorded. Alternatively 
or additionally, at least some demosaicing errors and/or interreflection errors 
and/or shadows present in the recorded image may be removed. Alternatively 
or additionally information relating to the physics of the scene, such as 
physical characteristics, may be provided. The transform function may be 
applied globally to the first image, or may be applied locally within the image. 
Information relating to which point or points within the image the transform 
function holds for may be used to provide information about the scene. 

The reference transforms may have been determined previously, using 
calibration as described above. 

According to the present invention there is also provided a method for 
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recording an image with image recording apparatus and processing the 
recorded image of a scene illuminated by an illuminant light, the method 
including calibration steps of: 

storing a digital response of an image recording apparatus to each of a 
plurality of colours of illuminant light E(X) f in an electrical memory means; 

grouping each colour of illuminant E a (X) into a pair with a different 
colour of illuminant E b (X), and for each pair of illuminants E a (X) and E b (X), 
calculating an illuminant transform function T ab , the transform function being 
the function which best maps the image recording apparatus response across 
the pair of illuminants, and recording the transfer function in an electrical 
memory means. 

The number of pairs of illuminants is advantageously the same as the 
number of distinct illuminants, the second illuminant in each pair being 
defined in terms of the first. Preferably the second illuminant is an optically 
filtered function of the first, such that E b (X) = E a (X)C(W, where E b (X) is the second 
illuminant, E a (X) is the first illuminant and C(X) defines the optical filter 
response as a funtion of wavelength X, Desirably the responses of the image 
recording apparatus are recorded for a set of distinct illuminants E(X). Further 
intermediate illuminants may be added through interpolation of the recorded 
responses. 

The method may also include the following image recording steps: 
recording the responses of the image recording apparatus to image light 

(P L ) from a scene to be recorded and to optically filtered image light (P 2 ) from 

the scene; 

determining which colour of illuminant E(X) is closest to the colour of the 
illuminating light, thereby estimating the colour of the illuminating light; and 

removing at least some colour bias due to the illuminating light from the 
recorded image and/or at least some of any demosaicing errors and/or 
interreflection errors. 
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The optically filtered image light is preferably filtered using a filter 
which produces an output which includes relatively more light of one 
wavelength than the input. The output may contain relatively more green light 
or more purple light than the input. 

The colour of the illuminating light is desirably determined by applying 
each transform function T ab to the recorded response (P : ) of the apparatus to 
the image light and comparing the transformed response (P ) to the response 
(P ) of the apparatus to the filtered image light, the transform function which 
best relates the two responses being the function which identifies the colour of 
the image light and the filtered image light. The best transform function is 
preferably the function which minimizes the error of the operation (T a,b P -P 2 ). 
The error is desirably the root mean square error. 

Embodiments of the invention will now be described in more detail by- 
way of example only with reference to the accompanying drawings, in which: 

Figure 1 is a diagrammatic representation of an image recording 
apparatus according to the invention; 

Figure 2 is a diagrammatic representation of an alternative image 
recording apparatus according to the invention; 

Figure 3 is a diagrammatic representation of a further alternative image 
recording apparatus according to the invention; and 

Figure 4 is a diagrammatic representation of a further alternative image 
recording apparatus according to the invention. 

Referring to figure 1 T a camera 10 includes a sensor array chip or CCD 12 
and an optical filter 14, located within a housing 20. The CCD 12 may be 
coated with the filter 14. The housing 20 is generally light-tight apart from an 
aperture 22 through which image light 16 from an object 0 enters the housing 
20. Part of the image light 16a passes through the filter 14 before striking the 
sensor array 12, while the rest of the image light 16b travels directly to the 
sensor array 12. First and second images relating to the light 16a and 16b 
respectively, are recorded by the sensor array 12 and the image data is 
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then passed to processing means 18 for processing to provide information 
regarding the colour of the illuminant light. This enables illuminant colour 
bias, demosaicing errors, or interreflection errors to be at least partly removed 
from the image, as described in more detail hereinafter. 

An alternative embodiment of the invention is shown in figure 2; the 
same reference numerals are used for corresponding features. Image light 16 
from the object O enters the housing 20, through aperture 22, in which optical 
processing means 24 and two sensor arrays 12 are located. The optical 
processing means 24 comprises an optical beamsplitter 26 and an optical filter 
14, provided in front of one sensor array 12a. On entering the housing 20, the 
light strikes the beamsplitter 26 and is split into two image beams 16a, 16b. 
One beam 16a passes through the filter 14 before striking one sensor array 12 a, 
while the other beam 16b is passed directly to the other sensor array 12b. An 
image of the object O is recorded on each sensor array 12a, 12b. The image 
data from each sensor array 12a, 12b is then passed to the processing means 18 
for processing, as described above. 

A further embodiment of the invention is shown in figure 3, in which the 
two sensor arrays 12a and 12b are located in separate housings 28, 30. A first 
part of the light 16a from the object O enters one housing 28 through aperture 
32, where it passes through an optical filter 14 before striking the sensor array 
12a. A second part of the light 16b enters the other housing 30 through 
aperture 34, where it passes directly to the sensor array 12b. The image of the 
object O is recorded by each sensor array 12 a, 12b. The image data from each 
sensor array 12a, 12b is then passed to the processing means 18 for processing, 
as described above. 

A further alternative embodiment of the invention is shown in Figure 4. 
In this embodiment, the camera 10 includes a single sensor array 12. The 
sensor array 12 includes six different types of pixels, responsive to respectively 
different frequencies of light. For example the array might include RGB pixels 
and R'G'B' pixels (where denotes filtering). A first image of the object 0 is 
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recorded by the pixels RGB and a second image of the object is recorded by the 
pixels R'G T B\ The image data is then passed to the processing means 18 for 
processing, as described above. 

The image data is processed in the processing means 18 according to the 
method for recording an image provided by the invention. 

In order to compensate for illuminant colour bias in an image recorded 
using image recording apparatus, such as a digital camera, the colour of the 
light E(X) illuminating the object 0 to be recorded is first estimated, as 
described below. 

The response of the camera sensor array 12 can, for example, be 
described by the following: 

U = LS(X) E(X) ¥{X) dX 

where X is wavelength, p is a 3-vector of the response of the image recording 
apparatus sensor array 12 (rgb pixel value), F is the 3-vector of response 
functions (red-, green- and blue-sensitive), E is the illumination striking a 
surface of reflectance S, and m is the visible wavelength spectrum. 

In order to solve this equation for E(X) one must have more known values 
than unknown values. In the present invention the image is recorded in two 
different wavelengths, by passing some of the image light 16a through a filter 
14, as described above. If there are k distinct surfaces on the object O then the 
number of measurements recorded by the sensor array 12 is 2*3* k and the 
number of unknowns in the above equation is 3 k +6 (assuming 3 dimensional 
linear models for the illumination and the surface). So long as there are 3 or 
more surfaces in the object O there will be more knowns than unknowns in the 
equation. 
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Assuming 3 surfaces in the object O, the above equation can be rewritten 

as 

[E L E 2 E3] = A(E l (X.))Iai22 23] 

where A(E L (X)) is a matrix with the z/th element equal to Le,(W E\X) S (X) dX , and 
E l (X) is unknown illuminant 1 (image light P . Under unknown illuminant 2 
(filtered image light P 2 ) the responses of the sensor array are equal to: 

ta/e; E3 , ] = a(e 2 (x))1 2i s> S3 ] 

The sensor array responses under the second illuminant can be expressed in 
terms of the responses under the first illuminant, through matrix inversion, as 
follows: 

In/ I>; £ 3 '1 = A(E 2 (X)) A(EW [E 1 E 2 B3] 

The illuminant transform matrix (T ab ) = A(E 2 (X)) A(E l {X))~ l encodes all the 
information necessary to solve for the unknown illuminants and E Z (X). 

The illuminant transform matrix is found by applying the method of the 
present invention. The digital response of the sensor array to each of a 
plurality' of known colours of illuminant light E(X) is recorded and stored in an 
electrical memory means. The response is recorded for each colour of 
illuminant light, E a (X), and for each colour of illuminant light when optically 
filtered, E b (X). Preferably a discrete set of colours of illuminant light are used. 
Preferably the chosen colours of illuminant light represent those that a scene 
might realistically be illuminated by, i.e. daylight at different times of the day t 
and internal light under a range of known lighting systems. These lights tend 
to include more red and blue light than green light. 

Each colour of illuminant E a (X) is paired with its optically filtered 
equivalent E b (X), and the illuminant transform matrix T ab calculated for each 
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pair of illuminants E a (X) and E b (X). The transform matrix which best maps the 
sensor array response across each pair of illuminants is selected, and recorded 
in an electrical memory means. 

Subsequently, the responses of the sensor array to the image light (P^ 
(unknown illuminant and to the optically filtered image light (P 2 ) 
(unknown illuminant E 2 (X)) t are recorded. The colour of the known illuminant 
E(X) closest to the colour of the unknown illuminating light is determined by 
applying each recorded transform matrix to the sensor array response under 
unknown illuminant EHW, to thereby estimate the sensor response under 
unknown illuminant E 2 (A,). The transform matrix which produces the estimated 
sensor response under unknown illuminant E 2 (X) which is closest to the 
recorded sensor response under unknown iUuminant E 2 (X), the filtered image 
light, is selected. The colour of the known illuminant E a (X) to which this 
transform function relates can be determined. This is the known illuminant 
E a (^) whose sensor response corresponds most closely to the sensor response, 
the unknown illuminant Thus the colour of the illuminating light can be 
estimated; the estimated illuminant light gives the closest sensor response to 
the sensor response of the unknown illuminant, but the spectral characteristics 
of the selected known illuminant may be different to the spectral characteristics 
of the unknown illuminant. Information relating to the known illuminant may 
be used to at least partially remove colour bias due to the illuminating light 
from the recorded image. The information may alternatively or additionally be 
used to remove at least some demosaicing errors and/or interreflection errors 
and/or shadows from the image. The information can further be used to obtain 
information relating to the physics of the scene, such as physical characteristics 
of the scene. 

It is desirable that the filter produces a wide range of different transform 
functions for different illuminants. Once a set of known illuminants has been 
chosen, a number of different possible filter functions may be applied to each 
illuminant and then the filtered and unfiltered responses compared to work out 
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a transform function. A filter is then chosen which produces the largest 
variation between transform functions for respectively different illuminants. 
The variance of the transform functions may be calculated using known 
techniques. 

One way of choosing the filter is as follows. Let us represent the 3 x N 
matrix of RGBs recorded under E a (X) as R a and the corresponding filtered RGBs 
as C af . The superscript f denotes a particular filter. The 3x3 matrix transform 
T a f that takes R a onto C af with minimum least-squares error is equal to: 



where [A] + = A^A'Af 1 is the Moore Penrose Inverse (a standard mathematical 
technique used to find a linear least-squares solution). Suppose we calculate 
the best transform matrix for a second filter f : 



Since light has the property of superposition, the camera response to two 
lights turned on together is equal to the sum of responses from each response 
turned on individually. If we interpret the filtering operation as changing the 
illumination, it follows that: 



From (1), (2) and (3) it is apparent that the linear sums of the filters lead 
to linear sums of transform matrices: 



T a .f = c aJ [R a ] + 



(1) 



jar = C a.f [R a ]+ 



(2) 




(3) 



C af [R a ] + + C r [R a ] + 



Let T u denote the transform matrix for illuminant i and filter j (i = 1, 2 t 
n) (j = 1, 2, m). The mean transform matrix for filter j is equal to: 
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v(T) = (5) 



Let M IJ = T lJ - ii(V): the transform matrix with the mean transform matrix 
for the given filter subtracted. For convenience we will now composite all of the 
n matrices (for a fixed filter j) M 1J into a vector v j . The vector v J is (3*3*n) x 1. 

In statistics, if x j is a scalar quantity denoting a population of m 
measurements and \x is the population mean, then variance is defined as: 



var(x) = (6) 



Of course when p. = 0 this simplifies to 



var(x)=^^- (7) 

n 



Grouping the n values x^ into anmxl vector x, then variance in (7) can be 
written as: 



var(x) = (x'x)/n (8) 

where (8) denotes a 1 x n matrix x l premultiplying a n x 1 matrix x using the 
usual rules of matrix algebra (8 is the dot product of two vectors). 

It follows then that the variance of transforms for filter j is equal to: 

var(T J ) - [v J lV (9) 



Let us now place the vectors y J (j = 1, 2, m) into the columns of a 9n x 
m matrix F (each column contains all the mean subtracted transforms for a 
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particular filter). By the property of light superposition we can generate the 
transform matrix vector for any linear combination of the m filters simply by 
adding together the columns of F. Suppose that c is a coefficient vector 
representing the particular combination of filters we wish to make. The 
corresponding transform vector is equal to Fc. It follows then that the 
transform variance for any filter (constructed by a linear combination of the m 
original filters) is equal to: 



In designing the best filter, we wish to find the coefficient vector c that 
maximises (10). Looking at (10) this is trivial to do, we simply make the 
elements in c as large as possible. To prevent this trivial solution we must fix 
the magnitude of c. 

Let G be a 9 x m matrix where the jth column is the 3x3 mean matrix 
jifF) stretched out as a vector. It follows then the c r G c Gc denotes the magnitude 
of the mean illuminant transform matrix. If we can fix this magnitude for any 
derived filter then we can find c which maximises (10). This is done by adding 
an appropriate Lagrange multiplier term to (10): 



To maximise (11) we must differentiate and equate to 0 in the usual way: 



var(T £ ) = cTTc 



(10) 



maximise I = c r FFc + Mc'G'Gc - 1) 



(ID 



Sc 



= 2FFc + XG z Gc = 0 



(12) 



where Q is the m x 1 zero vector. It is clear that: 



[G c Gr l FF£= ^c 



(13) 
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It follows that the c than maximizes (11) is an eigenvector of (13). This can be 
found in the usual way. 

Above, we added a constraint in order to fix the magnitude of the 
coefficient vector c. Specifically, we fixed the magnitude of the mean 
illuminant transform matrix. Other constraints could have been used. For 
example in carrying out the above minimization it is important that a large 
variance in transform matrices occurs in tandem with good colour correction. 
That is, there is no point in finding a filter that induces large transform 
variance, if the same filter breaks the linear model of illuminant change. It is 
important both that linear models accurately map RGBs across the filter 
correction and that there is a large variance in these transforms. 

We can in fact enforce this intuition if we change the constraint matrix G 
defined above. Specifically let each column of G contain the difference between 
corrected (i.e. least-squared fitted) RGBs and corresponding filtered RGBs. 
Remember T f R* - C f denotes the RGB difference between corrected RGBs and 
corresponding filtered RGBs for illuminant i and filter f. It follows then that 
the fth column of G contains T If R' - C' f (i = 1,2, n) stretched out (as discussed 
above) as a long vector. 

With G defined in this way, (11) seeks to maximise the variance of the 
transform matrices subject to the constraint that the magnitude of the error 
between corrected and corresponding filtered RGBs is held fixed. That is the 
ratio of transform variance over correction error is maximised. 

In the Figure 2 embodiment, the two sensor arrays are spatially 
registered so that it is possible to relate corresponding scene pixels. Thus, it is 
easy to take each illuminant transform matrix in turn and determine how well 
it relates the pixels recorded under the filtered and unfiltered lights. 

For the Figure 3 embodiment, the two cameras are mounted on a stereo 
head as part of an active vision system. To find corresponding points (and 
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therefore corresponding scene pixels) to test the transform matrices involves 
fusing the stereo images using existing available algorithms. 

It will be understood that in the Fig. 1 embodiment of the invention, a 
single sensor array simultaneously records the image light P i and the optically 
filtered image light P 2 . The central region of the array is struck by optically 
filtered light, while the remainder of the array is struck by unfiltered light. 
Thus, no single part of the scene to be recorded is registered in both filtered 
and unfiltered form. Because of this, finding corresponding scene pixels is 
more difficult than in the embodiments shown in Figures 2 and 3. It is 
necessary to identify regions of constant surface colour which intersect the 
boundary between the centre of the CCD (coated with the filter) and the 
adjacent uncoated region. This can be done using existing techniques. Such an 
analysis provides corresponding colours, with and without filtering, and as such 
provides the means to estimate the illuminant. Alternatively the camera may 
be moved so that the same part of a scene may be recorded both with and 
without the filter. 

A modification of the Figure 4 embodiment may be made, based on an 
extension of the ideas previously discussed. 

Since a change in illumination maps the RGB responses linearly, it 
follows that individual RGBs (just the red responses, just the green responses or 
just the blue responses) are mapped linearly (this is the definition of a linear 
transform). Let us focus our attention on the relationship between the original 
camera RGBs and the response of a single colour channel under filtering; that is 
let us consider how we might map 3 numbers into 1. Using the principles 
discussed above, a pre-processing step is carried out as follows: 

1. Measure a set of known illuminants E 1 (X), E 2 (k), E v (X). 

2. For each illuminant pair E a (X) and E b (X) calculate the illuminant 
transform vector V ab . This vector is the 1 x 3 matrix that best maps the RGB 
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camera responses recorded under illuminant a to the corresponding red (or 
green or blue) camera responses measured under illuminant b. By definition, 
V a b is the first (or second or third) row of T ab . 

Camera measurements are subsequently made under an unknown 
illuminant. 

1. P denotes the camera measurements under the unknown illuminant. 
Let p 2 denote the corresponding red (or green or blue) row vector of 
measurements under a second illuminant. 

2. For each plausible illuminant pair (a, b) calculate ||V ab P t - g 2 || (where ||.|| 
denotes root mean square error or similar). 

3. Return the transform vector V ab that minimises the error. Associated 
with this vector is the colour of the illuminant. 

Clearly the 'vector 5 algorithm is very similar to the original method. 
However, because a vector is composed of three numbers in contrast to 9 for a 
linear transform, we expect the simplified algorithm to be less discriminatory: 
illuminants will induce vectors that are more similar than corresponding 9 
parameter 3x3 matrices. Indeed, experiments bear out this observation. 
However, the vector-based algorithm leads to further simpler embodiments of a 
camera according to the invention. 

In the modification to the Figure 4 embodiment, the CCD includes four 
different types of pixels, each particularly responsive to different frequencies of 
light. In this embodiment the pixels are responsive to red, green and blue light 
and to a frequency of light (designated F) which optimises image processing. 

Three of the types of pixels (RGB) are selected to record the first image, 
and the fourth type of pixel (F) records the second image. No filter is used. 
Because the pixels F are responsive to different frequencies of light to RGB, 
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their response is equivalent (in terms of the method of the invention) to the 
optically filtered images of the previous embodiments. 

Thus, for each known illuminant the vector which maps the RGB 
response to the F response is established. 

Subsequently the response of the CCD to image light (an unknown 
illuminant) is recorded. The response of the RGB pixels is equivalent to the 
image light P 1 of the previous embodiments, while the response of the F pixels 
is equivalent to the optically filtered image light (P ) of the previous 
embodiments. 

The vector which best maps the RGB response to the F response is thus 
chosen from the set of calibration vectors. The unknown illuminant is then 
estimated as explained previously. 

As an alternative to the above, an existing four colour camera 
(comprising a mosaic of cyan, magenta, yellow and green responsive pixels) 
could be used. Alternatively, a specially designed three colour camera (RGF) is 
feasible. More than one CCD could be used to increase the information 
provided. 

There is thus provided an image recording apparatus and method which 
alleviates the problems caused by differing and unknown illuminant lights. 

It will further be appreciated that the present invention may additionally 
or alternatively be used to remove other image errors from an image recorded 
using image recording apparatus and to obtain other information about a scene 
from an image of the scene recorded using image recording apparatus. 

Various modifications may be made to the above embodiments while still 
falling within the scope of the invention. 
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In the embodiments which use two sensor arrays, the arrays may be 
arranged such that their red, green and blue responsive pixels correspond. 
Interpolation is preferably carried out before the two images are compared. 
Alternatively, the sensor arrays may be arranged such that, for example, a red 
responsive pixel on one array corresponds spatially to a green responsive pixel 
on the other array, to provide additional colour information at each pixel point. 

Alternatively, three sensing arrays may be used, as in a known "three 
chip" camera, with a filter in front of at least one array. This provides 
additional colour information. 

The method can be used to remove demosaicing errors, because the 
transform function should be applicable across the entire image. It can thus be 
used to detect and remove spurious coloured pixels, and to provide information 
concerning shadows and other artefacts caused by the physics of the scene 
being viewed. 

As a modification to the Figure 3 embodiment, two separate cameras 
each having a sensor array may be used, without a filter being provided in 
either camera. The cameras must have different responses such that the 
different optical processing of the two images results from the different camera 
responses. 

Although the use of filters is desirable, the filters may generally be 
replaced by the use of differently responsive pixels. Thus, instead of mapping 
an RGB response onto a filtered RGB response, the RGB response could be 
mapped onto the response of pixels which react to different colours. 

Whilst endeavouring in the foregoing specification to draw attention to 
those features of the invention believed to be of particular importance it should 
be understood that the Applicant claims protection in respect of any patentable 
feature or combination of features hereinbefore referred to and/or shown in the 
drawings whether or not particular emphasis has been placed thereon. 
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1. Image recording apparatus (10) for processing an image, the apparatus 
including: optical sensor means (12) for recording a first digital optical image 
of at least a part of a scene (0) illuminated by an illuminaut light and for 
recording a second digital optical image of at least a part of substantially the 
same scene under substantially the same Ulumlnanr light; the light producing 
the first and second images undergoing different optical processing; and means 
(18) in communication with the optical sensor means for processing 
information relating to the first and second images, wherein the processing 
means relates one of rhe first and second images to the other of the first and 
second images. 

2. Image recording apparatus (10) according to claim 1 wherein the 
processing means (18) correlates the first and second images, 

3. Image recording apparatus (10) according to any preceding claim wherein 
first and second optical sensor means (12) are provided for recording the first 
and second images respectively. 



4L Image recording apparatus (10) according to claim 3 wherein at least one 
of the first and second optical sensor means (12) is relatively broadband optical 
sensor means, being responsive to at least two distinct wavelengths of light 
within a broad spectrum of wavelengths. 

5. Image recording apparatus (10) according to claim 4 wherein the 
wavelengths are at least 100 nm apart* 

6> Image recording apparatus (10) according to any preceding claim wherein 
the optical sensor means (12) includes at least two types of optically sensitive 
elements, 
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responsive to respectively different wavelengths of light. 

7. Image recording apparatus according to any preceding claim wherein the 
optical sensor means (12) comprises a charge coupled device (CCD) chip, the 
chip comprising an array of photoelectric detector pixels. 

8. linage recording apparatus according to claim 7 wherein the pixels have a 
broad response centering on a particular wavelength of light. 

9. image recording apparatus according to claim 7 or claim 8 wherein the 
CCD chip is coated with a filter (14). 

10. Image recording apparatus according to any preceding claim wherein the 
optical processing means comprises an optical filter (14). 

11. Image recording apparatus according to claim 10 wherein the filter (14) 
has characteristics such that its output is linearly related to Its input. 

12. Image recording apparatus according to claim 10 or claim 11 wherein the 
response of the filter (14) is a smooth function with respect to wavelength and 
the filter (14) has an average transmittance of more than 308. 

13. image recording apparatus according to any of claims 10 to 12 wherein 
the Alter (14) produces an output which includes relatively mora light of one 
wavelength than of another wavelength as compared with the input. 

14. Image recording apparatus according to claims 10 to 13 wherein the filter 
(14) is located in the image light path before the optical sensor means. 

15. Image recording apparatus according to any preceding claim whereto 
first and second optical sensor means are provided by a single CCD chip (12) 
which records the first and second digital optical images. 
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16. Image recording apparatus according to claim 15 wherein die first and 
second sensor means may comprise respectively different parts of the chip (12). 

17. Image recording apparatus according to claim 15 or claim 16 wherein the 
first and second images comprise different parts of the image recorded by the 
CCD chip (12), in spatial terms or in terms of the frequencies of light recorded. 

18. Image recording apparatus according to any of claims 15 to 17 wherein a 
filter (14) Is provided in front of or on a parr of the CCD chip (12} such that the 
first or second digital optical image Is recorded by that part of the chip, and die 
other of the digital optical images is recorded by the remainder of the chip. 

19. Image recording apparatus according to any of claims 1 to 14 wherein 
the optical sensor means comprises a first CCD chip (12a) for recording the first 
digital optical image and a second CCD chip (12b) for recording the second 
digital optical image. 

20. Image recording apparatus according to claim 19 wherein the chips (12a, 
12b) are located in close proximity to one another, in the same geometric plane. 

21. Image recording apparatus according to claim 19 or claim 20 wherein the 
two CCD chips (12a, 12b) are responsive to respectively substantially the same 
frequencies of light, the optical processing means comprising an optical 
beamsplitter for splitting the image light into two parts and for directing each 
part of the light towards a respective one of the CCD chips, and an optical filter 
(14) being located in the path of one part of the image light, before one CCD 
chip. 

22. Image recording apparatus according to any of claims 1 9 to 22 wherein 
the optical sensor means and the optical processing means are located within a 
housing, such as a camera body. 

23. Image recording apparatus according to any of claims 19 to 21 wherein 
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each CCD chip (12a, 12b) ia provided in a separate housing, a first housing 
having a CCD chip (12b) provided therein and a second housing having a CCD 
chip (IZa) and an optical filter (14) provided therein. 

24« Image recording apparatus according to any of claims 19 to 21 wherein a 
first CCD chip is provided within a flrsr digital camera and a second CCD chip 
is provided within a second digital camera, such that die different optical 
processing of the two images result* from the different camera characteristics. 

23. Image recording apparatus according to any of claims 19 to 24 wherein 
the two chips are responsive to respectively different frequencies of light. 

26. Image recording apparatus according to any preceding claim wherein the 
processing means Is microprocessor based, having electrical memory means. 

27. Image recording apparatus according ro any preceding claim wherein the 
processing means includes means for providing information relating \o the 
spectral characteristics of the flluminant light 

28. Image recording apparatus according to claim 2 7 wherein information 
relating to the spectral characteristics of the illuminant light la used to facilitate 
removal of at least some of any Uluminant colour bias present fn the recorded 
image. 

29. Image recording apparatus according to claim 27 or claim 28 wherein the 
processing means includes means for facilitating the removal of at least some 
of any demoaaicing errors and/or intersection errors and/or shadows present 
in the recorded Image. 

30. Image recording apparatus according to any of claims 27 to 29 wherein 
the processing means includes means fox providing information relating to the 
physics of the scene, such as the physical characteristics of the scene. 
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31. A method for recording an image! the method including the steps of, 

(a) recording a first digital optical image of at least a parr of a scene 
illuminated by an iUuminant light and recording a second digital optical image 
of at least a parr of substantially the fame scene illuminated by substantially 
the same illuminant light; the light produdng the first and second images 
undergoing different optical processing; and 

(b) processing information relating to the first and second Images; 
wherein the processing step includes relating one of the first and second 
images to the other of the first and second images. 

32. A method for recording an image according to claim 3 1 wherein the first 
and second images are correlated, 

33. A method for recording an image according to any of claims 30 to 32 
wherein different optical processing results at least partly from the filtering of 
light produdng the first or second image. 



34. A method for recording an image according to any of claims 30 to 33 
wherein the different optical processing is provided by the use of sensors 
responsive to respectively different frequencies of light in recording the first 
and second images. 

35. A method for recording an image according to any of claims 30 to 34 
wherein the first and second images comprise respectively different parts of a 
global image of a scene. 

36. A method for recording an image according to any of claims 30 to 35 
wherein the processing of die information relating to the first and second 
images provides an estimate of the spectral characteristics of the iUuminant 
light. 



37. A method for calibrating image recording apparatus, the method being 
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according to any of claims 30 to 36. 

38. A method according to claim 3 7 wherein the method includes the 
carrying out of steps (a) and (b) for each of a plurality of different Known 
illuminant lights and wherein step (W includes the step of processing the 
information relating to the first and second images to provide an indication of 
the relationship therebetween. 

39* A method according to claim 36 wherein the indication of the 
relationship is a transform function) which may he a transform matrix, and the 
method provides a set of reference transform functions, each transform 
fraction relating to a different known Uluminant light* 

40. A method according to any of claims 30 to 36 for processing an image 
recorded using Image recording apparatus wherein the first and second images 
relate to a scene illuminated by an unknown illuminant. 

41. A method according to claim 40 wherein the method includes the step of 
applying one or more of the reference transform functions to the first or second 
image and determining the reference transform function which beat relates the 
two images. 

42. A method according to claim 41 wherein each reference transform 
function is applied to the first Image to produce a transformed first image, 
which is subsequently compared to the second image and the reference 
transform function which produces a transformed first image most closely 
resembling the second image is selected es the best reference transform 
function. 

43. A method according to claim 42 wherein the known illuminant light to 
which the best reference transform function relates is determined! to provide 
information relating to the spectral characteristics of the light illuminating the 
scene to be recorded. 
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44. A method according to claim 43 wherein at least some of tile colour bias 
due to the illuminating light is removed from the image of the scene to be 
recorded and/or at least some demosairing errors and/or taferreflection errors 
and/or shadows present in the recorded Image are removed 

45. A method for recording an Image with. Image recording apparatus and 
processing the recorded Image of a scene Illuminated by an illuminant light, the 
method Including calibration steps oft 

storing a digital response of an image recording apparatus to each of a 
plurality of colours of illuminant light E(W, in an electrical memory means; 

grouping each colour of illuxnlnant F(X) into a pair with a different 
colour of illuminant EtX), and for each pair of illuminants E*(X) and &{X), 
calculating an illuminant transform function T* g the transform function being 
the function which best maps the image recording apparatus response across 
the pair of illuminants, and recording the transfer function in an electrical 
memory means. 

46. A method according to claim 45 wherein the number of pairs of 
illuminants is the same as the number of distinct illuminants, the second 
illuminant in each pair being defined In terms of the first. 

47. A method according to claim 45 including the step of recording the 
responses of the image recording apparatus for a set of distinct illuminants 
E(X). 

48. A method according to any of claims 45 to 4 7» the method further 
including the following image recording steps; 

recording the responses of the image recording apparatus to image light 
(P t ) from a scene to be recorded and to optically filtered image light (P^ from 
the scene; 

determining which colour of illuminant E(X) is closest to the colour of th<*< 
illuminating light, thereby estimating the colour of the illuminating light; and 
removing at least some colour bias due to the illuminating light from the 
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recorded Image and/or at least some of any demosaidng errors and/or 
lac errefleetion errors. 

49. A method according to claim 48 wherein the optically filtered image light 
is filtered using a filter which produces an output which includes relatively 
more light of one wavelength than the input 

50. A method according to claim 48 or claim 49 wherein the colour of the 
illuminating light is determined by applying each transform function T** to the 
recorded response (P ( ) of the apparatus to the image light and comparing the 
transformed response CP) to the response (?) of the apparatus to the filtered 
image light, the Transform function which best relates the two responses being 
the function which identifies the colour of the image light and the filtered 
image light* 

51. A method according to claim 50 wherein the best transform function is 
defined as the function which minimises the error of the operation CT**, -PJ. 

52. Image recording apparatus substantially as herein described with 
reference to Fig. 1 of the drawings,, 

53. Image recording apparatus substantially as herein described with 
reference to Fig. 2 of rhe drawings, 

54. finage recording apparatus substantially as herein described with 
reference to Fig. 3 of the drawings. 

53, A method substantially as herein described with reference to the 
drawings. 

56, Any novel subject matter or combinion including novel subject marrer 
disclosed herein, whether or not within the scope of or relating to the same 
invention as any of the preceding claims. 
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Abstract 

An image recording apparatus for processing an image 
includes a camera (10) , or other optical sensor means, for 
recording a first digital optical image of at least part of a 
scene illuminated by an illuminant light and for recording a 
second digital optical image of at least a part of 
substantially the same scene under substantially the same 
illuminant light. The images may be recorded on respectively 
different sensor arrays (12a and 12b) . The light producing 
the first and second images undergoes different optical 
processing in order to produce those images, and processing 
means (18) in communication with the optical sensor means is 
provided for processing information relating to the first and 
second images. By relating the two images, the processing 
means (18) is able to provide information regarding the 
colour of the illuminate light or other physical properties 
of the illuminated scene. 
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A3 a below named inventor, I hereby declare that; 

My residence, post office address 4 and citizenship are as 
stated below next to my name. I believe I am the original, 
first, and sole inventor (if only one name is listed below) 
or en original, first and joint inventor (if plural names 
are listed below) of the subject matter which is claimed and 
for which a patent is sought on the invention <snt it-led; 

COLOUR IMAGE PICKUP APPARATUS 
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the claims, as amended by any amendment referred to above. 

I acknowledge the duty to disclose to the Patent and 
Trademark Office all information known to me to be material 
to patentability as defined in Title 37 , Code of Federal 
Regulations, Sec. 1.56. 

I have identified below any foreign application (s) for 
patent or inventor's certificate or any PCT international 
application (s> designating at least one country other than 
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application and of which I claim foreign priority benefits 
under Title 35, United States Code, Sec. 119, and I have 
also identified below any foreign application (s) for patent 
or inventor's certificate or any PCT international 
application (s) designating at least one country other than 
the United States of America filed by me on the same subject 
matter having a filing date before that of the 
application (s) of which priority is claimed. 
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Ab a named inventor, I hereby appoint the practitioners 
associated with Customer Nus&er 007812 (John Smith-Hill, 
Reg, Mo- 27,730 and Daniel J. Bedell, Reg. No. 30,156) to 
prosecute this application and transact all business in the 
Patent and Trademark Office connected therewith and in 
connection with the resulting patent . 
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wi tfiCufl tonifir NC^er^WTffl^r j 
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resulting patent from SWINDELL & PEARSON. 
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information and belief ar<= believed to be true; and further 
that these statements were made with the knowledge that 
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any patent issued thereon. 
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